The dplyr and tidyr packages

Pipe operator, %>%

• A tidyverse feature (magrittr package)
• Written as: %>% (percent greater-than percent)
• exprA %>% exprB evaluates expression A, and then sends its output to expression B as input

my_values <- c(1.33, 1.66, 2.33)

mean(my_values) %>% round(1)

# is the same as:

round(mean(my_values), 1)

Note that the first argument of round() has disappeared in the piped version, because it is filled by the mean just calculated. 1 is the digits argument, i.e., one decimal place.

Keyboard shortcuts revisited

filter() subsets cases

• filter(df, ...) where df is the data frame and ... are 1+ logical expressions; each case that tests TRUE to the condition(s) will appear in the output
  • Logical expression means that the expression returns TRUE or FALSE when evaluated
• For the operations on the next slide, remember they are vectorized—the comparison is made to each value in the vector, and a vector of equal length is returned.

Sidebar: missing data (NA)

Missing data or the concept of missingness is represented in R with the symbol NA (not the string "NA" in quotation marks), for “Not Available”. It is equivalent to an empty cell in Excel.

select() and rename() work on variables

• select(df, v1, v2, v3), where v1 etc. are variable names, is for selecting which variables you want to retain in the data frame.
  • Numbers also work as an alternative to names
  • Use a negative sign ( - ) to negate a column (i.e., “all variables except…”)
  • Note: dropped variables are removed from the data frame, not merely hidden!
• rename(df, new_name = old_name) renames a variable old_name to new_name

# select the species column, and cols 4 through 8 except for 5:
penguins %>% select(species, 4:8, -5)

# rename species to "common_name"
penguins %>% rename(common_name = species)

mutate() creates a new variable in the data frame

• mutate(df, new_variable = expr) creates new_variable in df
• expr may be: a mathematical expression, a function call, a vector of appropriate length, or a fixed value. expr may also implement “if-then” logic, using one or more variables of the same case (e.g., “If temperature is above 90, then heat category is High”).

# isolate body masses, then convert penguin mass from g to kg and lbs:
penguins %>% 
  select(body_mass_g) %>% 
  mutate(body_mass_kg = body_mass_g / 1000,
         body_mass_lbs = body_mass_g / 453.6)

# to store the result back to penguins:
penguins %>% 
  mutate(body_mass_kg = body_mass_g / 1000,
         body_mass_lbs = body_mass_g / 453.6)

Sidebar: factors

A common mutate() task is to convert an existing variable’s type or measurement units. For example, our penguins_raw$Island variable is encoded as character data, but we would like to convert it to a categorical variable. In R, a categorical variable is encoded in a factor, a vector which accepts only certain values. Factors may be ordered.

# penguins_raw is a version of penguins without factors encoded

summary(penguins_raw$Island) # no level counts

# convert Island to factor, and assign result back to penguins_raw:
penguins_raw <- penguins_raw %>% 
  mutate(Island = as_factor(Island))

summary(penguins_raw$Island) # levels with counts!

summarize() returns a single row of summary calculations

• Aggregation functions
  • center: mean(), median()
  • spread: sd(), IQR()
  • range: min(), max()
  • position: first(), last(), nth()
  • count: n(), n_distinct()
  • logical: any(), all()
• To name the columns in the output data frame, use named arguments.
• Usually we will summarize after grouping (next section)

group_by() creates groupings using a variable

• group_by(v1) groups cases in the data according to their value for the variable (factor) v1.
• Grouping information is appended to the data frame as metadata
• summarize() understands these groups and applies function calls to the groups, rather than the whole data set, returning one row for each group
• group_by(v1, v2) groups cases by v1 and then v2 (order does matter)

pivot_longer() collapses 3+ columns into two

…creating more rows (a “longer” data frame)

pivot_longer(cols, names_to, values_to) where

• cols is a vector of variable names (or selection such as using across()), whose columns you want to collapse
• names_to is the name of a new variable which will receive the names of the collapsing columns
• values_to is the name of a new variable which will receive the values of the collapsing columns

pivot_longer() example: religion and income

tidyr’s relig_income dataset, from a Pew religion and income survey, has 1 row per religion and a column for the count of people in each income category. Let’s treat each count as its own observation. (This and following example are from the tidyr Pivoting vignette, also available by running vignette("pivot").)

data(relig_income)
names(relig_income)
# sample 5 random rows:
slice_sample(relig_income, n = 5)

relig_income_long <- relig_income %>% 
  pivot_longer(cols = 2:11,
               names_to = "income_category",
               values_to = "count")

slice_sample(relig_income_long, n = 5)

pivot_wider() example: fish encounters

It’s relatively rare to need pivot_wider() to make tidy data, but it’s often useful for creating summary tables for presentation, or data in a format needed by other tools. The fish_encounters dataset, contributed by Myfanwy Johnston, describes when fish swimming down a river are detected by automatic monitoring stations. Many tools used to analyse this data need it in a form where each station is a column.

data(fish_encounters)
names(fish_encounters)
# sample 5 random rows:
slice_sample(fish_encounters, n = 5)

fish_encounters_wide <- fish_encounters %>% 
  pivot_wider(names_from = station,
              values_from = seen)

# use glimpse() for previewing a wide table
slice_sample(fish_encounters_wide, n = 5) %>% 
  glimpse()

left_join() example: penguin species

Let us supplement our penguin observations with a table of information about the penguin species. Note that not all of our observed species in x exist in y. Note also that there is a species in y that we have not observed in x.

species_info <- read_excel("data/penguin-species.xlsx")
species_info
penguins %>% 
  left_join(y=species_info, by="species")

Other tabular combinations

• More joins
  • right_join() treats the right df as the primary table, keeping all its rows
  • inner_join() returns the minimal set, because it requires values in both tables
  • full_join() returns the maximal set, keeping all rows from both df’s
• Row- and column-binding: combining tables non-relationally (bind_rows(), bind_cols()
• Set operations
  • union
  • intersect
  • setdiff